feat: skip hash shuffle for date_bin/date_trunc on Range([timestamp]) by NGA-TRAN · Pull Request #24501 · apache/datafusion

NGA-TRAN · 2026-08-19T19:55:06Z

Which issue does this PR close?

Closes Support functions date_bin and date_trunc in range partition satisfaction #23569
Depends on test: add range-partitioned sorted time-bin aggregation coverage #24500 (test coverage that pinned today's hash-shuffle plan)

Rationale for this change

A table that is range-partitioned on timestamp (for example by hour) and grouped by date_bin / date_trunc of that timestamp is already partition-disjoint when bins do not straddle split points. DataFusion still planned Partial → hash RepartitionExec → Final.

After this change, Range([timestamp]) subset-satisfies KeyPartitioned([key, f(timestamp)]) when f is monotonic (date_bin, date_trunc) and evaluating f at each split vs. its predecessor shows the bins are disjoint. Aggregation then runs as one streaming SinglePartitioned step with ordering_mode=Sorted.

Bins that do straddle the split (for example date_trunc('day') on hour-partitioned data) still hash-repartition.

#24500's expected plan changes from:

AggregateExec FinalPartitioned
  RepartitionExec Hash([key, date_bin(...)])
    AggregateExec Partial, ordering_mode=Sorted
      DataSourceExec Range([timestamp]), 2 file groups

to:

AggregateExec SinglePartitioned, ordering_mode=Sorted
  DataSourceExec Range([timestamp]), 2 file groups

What changes are included in this PR?

Project Range through monotonic grouping expressions when bins stay disjoint.
Treat that case as subset satisfaction so SanityCheckPlan accepts SinglePartitioned over Range([timestamp]).
Update range_sorted_time_bin_agg.slt for date_bin(60s) and date_trunc('hour') (no shuffle) and date_trunc('day') (shuffle remains).

Are these changes tested?

Unit tests in partitioning.rs for aligned date_bin / date_trunc('hour') vs unaligned split and date_trunc('day').
cargo test --test sqllogictests -- range_sorted_time_bin_agg.slt

Are there any user-facing changes?

Queries that group by date_bin / date_trunc of a range-partitioned timestamp may skip a hash shuffle when bins do not cross file-group boundaries. Results are unchanged.

Test plan

cargo test -p datafusion-physical-expr --lib -- partitioning
cargo test --test sqllogictests -- range_sorted_time_bin_agg.slt
CI

Co-author with Cursor

Pin today's Partial + hash RepartitionExec + Final plan for GROUP BY key, date_bin(timestamp) on a table that is already Range([timestamp]) and sorted on (key, timestamp), so a follow-up can remove the shuffle. Co-authored-by: Cursor <cursoragent@cursor.com>

codecov-commenter · 2026-08-19T20:23:45Z

Codecov Report

❌ Patch coverage is 95.17241% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.46%. Comparing base (8332cfa) to head (ec91625).
⚠️ Report is 55 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/physical-expr/src/partitioning.rs	95.10%	20 Missing and 14 partials ⚠️
...on/physical-expr/src/equivalence/properties/mod.rs	90.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24501      +/-   ##
==========================================
+ Coverage   81.32%   81.46%   +0.14%     
==========================================
  Files        1117     1118       +1     
  Lines      396269   400318    +4049     
  Branches   396269   400318    +4049     
==========================================
+ Hits       322260   326130    +3870     
+ Misses      55186    55174      -12     
- Partials    18823    19014     +191

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Use generic column and table names so the coverage does not expose a metrics-specific schema. Co-authored-by: Cursor <cursoragent@cursor.com>

Accept RecordBatches and optional file sort order so the time-bin table reuses the same registration path. Co-authored-by: Cursor <cursoragent@cursor.com>

Treat Range([x]) as a subset of KeyPartitioned([..., f(x), ...]) when f is monotonic and bins do not straddle split points, so GROUP BY key, date_bin/date_trunc can stream without a hash RepartitionExec. Co-authored-by: Cursor <cursoragent@cursor.com>

Drop the local order-preserving walk and scalar predecessor so date_bin/date_trunc range projection uses EquivalenceProperties and interval_arithmetic instead of duplicating them. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

…che#24500 Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

NGA-TRAN · 2026-08-20T20:16:36Z

Tag @gene-bordegaray and @jayshrivastava for review

jayshrivastava

Looks good. Far better than my approach in #23536.

I left some comments re. completeness.

gene-bordegaray

all my comments largely ditto @jayshrivastava I can take a review when these are addressed 👍

…atisfaction Require matching SortOptions so -x is not treated as monotonic, try every monotonic projection candidate, and check date_bin disjointness even when subset satisfaction is disabled. Co-authored-by: Cursor <cursoragent@cursor.com>

NGA-TRAN

@jayshrivastava & @gene-bordegaray
I have addressed all Jayant's comments

Range([x]) grouping by a split-aligned date_bin/date_trunc of x is Exact when that transform is the whole key, and Subset only when extra keys remain and subset satisfaction is allowed. Co-authored-by: Cursor <cursoragent@cursor.com>

Add unit coverage for empty keys, multi-split disjointness, and evaluate/project error paths, plus SLTs for GROUP BY date_bin/date_trunc without extra grouping columns. Co-authored-by: Cursor <cursoragent@cursor.com>

NGA-TRAN mentioned this pull request Aug 19, 2026

Support functions date_bin and date_trunc in range partition satisfaction #23569

Open

github-actions Bot added physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) labels Aug 19, 2026

NGA-TRAN and others added 3 commits August 20, 2026 10:27

test: generalize range-sorted time-bin aggregation test names

58dea62

Use generic column and table names so the coverage does not expose a metrics-specific schema. Co-authored-by: Cursor <cursoragent@cursor.com>

test: share parquet listing-table helper for range tests

6006096

Accept RecordBatches and optional file sort order so the time-bin table reuses the same registration path. Co-authored-by: Cursor <cursoragent@cursor.com>

NGA-TRAN force-pushed the feat/issue-23569-range-date-bin-trunc branch from 05705f6 to b26846d Compare August 20, 2026 15:44

github-actions Bot added the logical-expr Logical plan and expressions label Aug 20, 2026

NGA-TRAN and others added 2 commits August 20, 2026 14:47

fix: wrap Range([x]) in backticks to satisfy rustdoc link checks

693cd3e

Co-authored-by: Cursor <cursoragent@cursor.com>

test: cover range date_bin/date_trunc edge cases and checked_predecessor

c36301c

Co-authored-by: Cursor <cursoragent@cursor.com>

NGA-TRAN marked this pull request as ready for review August 20, 2026 19:16

NGA-TRAN and others added 2 commits August 20, 2026 15:34

merge origin/main to pick up range-sorted time-bin agg tests from apa…

02029fb

…che#24500 Co-authored-by: Cursor <cursoragent@cursor.com>

test: fail closed when date_bin range splits are null or invalid

0c118e0

Co-authored-by: Cursor <cursoragent@cursor.com>

xavlee mentioned this pull request Aug 21, 2026

feat: stream aggregates for group-contiguous input #24497

Draft

jayshrivastava approved these changes Aug 21, 2026

View reviewed changes

jayshrivastava mentioned this pull request Aug 21, 2026

consider date_bin in range partition satisfaction #23536

Closed

gene-bordegaray reviewed Aug 22, 2026

View reviewed changes

NGA-TRAN mentioned this pull request Aug 25, 2026

Support date_bin / date_trunc satisfaction for multi-key Range partitioning #24644

Open

NGA-TRAN commented Aug 25, 2026

View reviewed changes

jayshrivastava reviewed Aug 25, 2026

View reviewed changes

Comment thread datafusion/physical-expr/src/partitioning.rs

Comment thread datafusion/physical-expr/src/partitioning.rs Outdated

NGA-TRAN and others added 2 commits August 25, 2026 14:42

test: cover remaining date_bin range Exact/Subset cases

ec91625

Add unit coverage for empty keys, multi-split disjointness, and evaluate/project error paths, plus SLTs for GROUP BY date_bin/date_trunc without extra grouping columns. Co-authored-by: Cursor <cursoragent@cursor.com>

Uh oh!

Conversation

NGA-TRAN commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Test plan

Uh oh!

codecov-commenter commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

NGA-TRAN commented Aug 20, 2026

Uh oh!

jayshrivastava left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gene-bordegaray left a comment

Choose a reason for hiding this comment

Uh oh!

NGA-TRAN left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

NGA-TRAN commented Aug 19, 2026 •

edited

Loading

codecov-commenter commented Aug 19, 2026 •

edited

Loading